AirLLM is a framework that streams model layers from disk one at a time during inference, letting you run massive language models like DeepSeek-V3 (671B parameters) on single consumer GPUs without quantization or distillation. It also supports fine-tuning huge models on small VRAM by keeping only adapters in memory while streaming frozen weights.

• Runs a 671B parameter model on ~12GB VRAM and 125B models on ~6GB by keeping only one decoder layer on GPU at a time, with the rest streamed from disk
• Works with virtually every major open-source model family (Llama, Qwen, DeepSeek, Mistral, Gemma, etc.) through a single `AutoModel.from_pretrained()` call with no special configuration
• Supports optional 4-bit/8-bit weight quantization for up to 3x speedup, and fine-tuning of 125B models under 6GB VRAM by streaming base weights while keeping adapters resident